feat(stats): wire token-savings telemetry into search and find_related - #82

…y) into incremental reindexing

…vectors, and BM25 postings (#91)

* feat(index): incremental reindexing — reuse unchanged files' chunks, vectors, and BM25 postings

Port upstream semble #225 (partial reindexing) to the Rust core. When the
cached index's whole-tree content hash is stale, `load_or_build_index` now
seeds the rebuild with the previous index instead of rebuilding from
scratch: files whose per-file content hash is unchanged keep their chunks,
vector rows, and BM25 postings; only changed files are re-chunked and
re-embedded, and deleted files' postings are dropped.

- `indexing/types.rs`: `FileManifestEntry {hash, start, count}`,
  `PreviousIndex::try_new` (alignment checks), `make_chunk_id`.
- `sparse.rs`: `Bm25Index` becomes the id-keyed incremental index from
  upstream `bm25.py` (`add_document` / `remove_document` /
  `set_doc_order`); `bm25.json` v2 persists `{documents, docOrder}`.
- `create.rs`: `create_index_from_path(.., previous)` reuse path; rows
  are moved (not copied) and reused rows are not re-normalised.
- `cache_orchestrator.rs`: `load_previous_for_incremental` (fails closed
  on any structural inconsistency) + shared `manifest_compatible`.
- `index.rs`: `files` manifest in `IndexManifest`/`CspIndex`,
  `from_path_with_previous`, `INDEX_SCHEMA_VERSION` 1 → 2, and
  `load_from_disk` rejects component count mismatches.
- ADR-0005 records the per-file content hash (vs upstream `mtime_ns`)
  decision; `semble.md` and both READMEs updated.

Refs #84

* fix(index): harden incremental reindex after review

- `PreviousIndex::try_new`: sort manifest entries by `(start, count)` so a
  zero-chunk file that ties with the following file no longer fails the
  tiling check (which silently disabled incremental reuse for that tree).
  Regression test `zero_chunk_file_does_not_break_manifest_tiling`.
- `Bm25Index::load`: rebuild postings from the persisted term counts via
  `insert_document` instead of materialising `freq` copies of every term;
  sum lengths in u64 and reject out-of-range counts. Drop the duplicate
  `Doc.chunk_id`.
- `create_index_from_path`: embed all changed files' chunks in one batched
  pass (`dense::embed_chunk_refs`) so a cold build keeps the tokenizer's
  batch parallelism.
- `FileManifestEntry::end()`: saturating add so a corrupt manifest fails
  the range checks instead of overflowing.
- `load_previous_for_incremental`: reject a seed whose vector rows do not
  match the live model's dimension, falling back to a full rebuild.
- `parse_manifest`: read `files` through the `FileManifestEntry` serde
  derive that `save` writes with.
- Docs: query-term de-duplication is a real ranking divergence from
  upstream's query-frequency weighting, not rank-neutral; record it as an
  open parity gap in ADR-0005 and `semble.md`.

Refs #84

* fix(index): skip files whose lossy display path collides with an indexed file

On Unix, file names that differ only in invalid UTF-8 bytes collapse to
the same `to_string_lossy` path. The BM25 chunk ids derived from that
path would then collide and abort the whole build with
"chunk_id already indexed". Keep the first such file, skip the rest
with a warning, and add a Linux-only regression test (APFS rejects
non-UTF-8 names).

Refs #84

* chore: merge origin/main (#80 max_snippet_lines, #82 savings telemetry) into incremental reindexing

* fix(index): reject zero BM25 term counts on load; take persisted vectors verbatim

- Bm25Index::load rejects a zero term frequency (it would inflate the
  term's document frequency) so the cache falls back to a full rebuild.
- SelectableBasicBackend::load no longer re-normalises rows that were
  normalised before save, keeping unchanged rows bit-identical across an
  incremental rebuild seeded from disk.

Refs #84

* refactor(index): split create/sparse tests out, extract create_index_from_path helpers

- create.rs / sparse.rs test modules move to create/tests.rs and
  sparse/tests.rs, matching the index/, dense/, cache_orchestrator/ layout.
- create_index_from_path delegates to open_previous, display_path,
  take_previous_rows and embed_fresh_rows; behaviour unchanged.
- load_previous_for_incremental compares the content selection as a set,
  so a duplicated request no longer matches a manifest that covers more.

Refs #84

* perf(index): compare the cached backend dim instead of scanning every row

Refs #84

* test(index): build the manifest key with the platform separator

Refs #84